<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Custom deep learning layers support</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d9/df8/tutorial_root.html">OpenCV Tutorials</a></li><li class="navelem"><a class="el" href="../../d2/d58/tutorial_table_of_content_dnn.html">Deep Neural Networks (dnn module)</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Custom deep learning layers support </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p><b>Prev Tutorial:</b> <a class="el" href="../../d5/d86/tutorial_dnn_javascript.html">How to run deep networks in browser</a></p>
<p><b>Next Tutorial:</b> <a class="el" href="../../d9/d1e/tutorial_dnn_OCR.html">How to run custom OCR model</a></p>
<table class="doxtable">
<tr>
<th align="right"></th><th align="left"></th></tr>
<tr>
<td align="right">Original author </td><td align="left">Dmitry Kurtaev </td></tr>
<tr>
<td align="right">Compatibility </td><td align="left">OpenCV &gt;= 3.4.1 </td></tr>
</table>
<h2>Introduction</h2>
<p>Deep learning is a fast growing area. The new approaches to build neural networks usually introduce new types of layers. They could be modifications of existing ones or implement outstanding researching ideas.</p>
<p>OpenCV gives an opportunity to import and run networks from different deep learning frameworks. There are a number of the most popular layers. However you can face a problem that your network cannot be imported using OpenCV because of unimplemented layers.</p>
<p>The first solution is to create a feature request at <a href="https://github.com/opencv/opencv/issues">https://github.com/opencv/opencv/issues</a> mentioning details such a source of model and type of new layer. A new layer could be implemented if OpenCV community shares this need.</p>
<p>The second way is to define a <b>custom layer</b> so OpenCV's deep learning engine will know how to use it. This tutorial is dedicated to show you a process of deep learning models import customization.</p>
<h2>Define a custom layer in C++</h2>
<p>Deep learning layer is a building block of network's pipeline. It has connections to <b>input blobs</b> and produces results to <b>output blobs</b>. There are trained <b>weights</b> and <b>hyper-parameters</b>. Layers' names, types, weights and hyper-parameters are stored in files are generated by native frameworks during training. If OpenCV mets unknown layer type it throws an exception trying to read a model:</p>
<div class="fragment"><div class="line">Unspecified error: Can&#39;t create layer &quot;layer_name&quot; of type &quot;MyType&quot; in function getLayerInstance</div></div><!-- fragment --><p>To import the model correctly you have to derive a class from <a class="el" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html" title="This interface class allows to build new Layers - are building blocks of networks. ">cv::dnn::Layer</a> with the following methods:</p>
<div class="fragment"><div class="line"><span class="keyword">class </span>MyLayer : <span class="keyword">public</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html">cv::dnn::Layer</a></div><div class="line">{</div><div class="line"><span class="keyword">public</span>:</div><div class="line">    MyLayer(<span class="keyword">const</span> <a class="code" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html">cv::dnn::LayerParams</a> &amp;<a class="code" href="../../d1/dae/namespacecv_1_1gapi_1_1ie.html#a3ab1729bcaf2d08e30dd2bc645410908">params</a>);</div><div class="line"></div><div class="line">    <span class="keyword">static</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga6395ca871a678020c4a31fadf7e8cc63">cv::Ptr&lt;cv::dnn::Layer&gt;</a> create(<a class="code" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html">cv::dnn::LayerParams</a>&amp; <a class="code" href="../../d1/dae/namespacecv_1_1gapi_1_1ie.html#a3ab1729bcaf2d08e30dd2bc645410908">params</a>);</div><div class="line"></div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">bool</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a3405ff43150088f3cc1d54f7e9b3ec29">getMemoryShapes</a>(<span class="keyword">const</span> std::vector&lt;std::vector&lt;int&gt; &gt; &amp;inputs,</div><div class="line">                                 <span class="keyword">const</span> <span class="keywordtype">int</span> requiredOutputs,</div><div class="line">                                 std::vector&lt;std::vector&lt;int&gt; &gt; &amp;outputs,</div><div class="line">                                 std::vector&lt;std::vector&lt;int&gt; &gt; &amp;internals) <span class="keyword">const</span> <a class="code" href="../../db/de0/group__core__utils.html#ga4d89d63e402ef9ddc48e18e21180fe4a">CV_OVERRIDE</a>;</div><div class="line"></div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">void</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a39236126b72a365b94b132abd5bd946b">forward</a>(<a class="code" href="../../d4/d32/classcv_1_1__InputArray.html">cv::InputArrayOfArrays</a> inputs,</div><div class="line">                         <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> outputs,</div><div class="line">                         <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> internals) <a class="code" href="../../db/de0/group__core__utils.html#ga4d89d63e402ef9ddc48e18e21180fe4a">CV_OVERRIDE</a>;</div><div class="line"></div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">void</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a070e355e782f14f97879f18747fd70fc">finalize</a>(<a class="code" href="../../d4/d32/classcv_1_1__InputArray.html">cv::InputArrayOfArrays</a> inputs,</div><div class="line">                          <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> outputs) <a class="code" href="../../db/de0/group__core__utils.html#ga4d89d63e402ef9ddc48e18e21180fe4a">CV_OVERRIDE</a>;</div><div class="line">};</div></div><!-- fragment --><p> And register it before the import:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="../../df/d8c/layer_8details_8hpp.html">opencv2/dnn/layer.details.hpp</a>&gt;</span>  <span class="comment">// CV_DNN_REGISTER_LAYER_CLASS</span></div><div class="line"></div><div class="line"><span class="keyword">static</span> <span class="keyword">inline</span> <span class="keywordtype">void</span> loadNet()</div><div class="line">{</div><div class="line">    <a class="code" href="../../df/d8c/layer_8details_8hpp.html#a7e8d9c0c5849b6a081ba2a84845f3dac">CV_DNN_REGISTER_LAYER_CLASS</a>(Interp, InterpLayer);</div><div class="line">    <span class="comment">// ...</span></div></div><!-- fragment --> <dl class="section note"><dt>Note</dt><dd><code>MyType</code> is a type of unimplemented layer from the thrown exception.</dd></dl>
<p>Let's see what all the methods do:</p>
<ul>
<li>Constructor</li>
</ul>
<div class="fragment"><div class="line">    MyLayer(<span class="keyword">const</span> <a class="code" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html">cv::dnn::LayerParams</a> &amp;<a class="code" href="../../d1/dae/namespacecv_1_1gapi_1_1ie.html#a3ab1729bcaf2d08e30dd2bc645410908">params</a>);</div></div><!-- fragment --><p> Retrieves hyper-parameters from <a class="el" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html" title="This class provides all data needed to initialize layer. ">cv::dnn::LayerParams</a>. If your layer has trainable weights they will be already stored in the Layer's member <a class="el" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a9a5578e0b3a0ec0301fb7320b54aa6ed" title="List of learned parameters must be stored here to allow read them by using Net::getParam(). ">cv::dnn::Layer::blobs</a>.</p>
<ul>
<li>A static method <code>create</code></li>
</ul>
<div class="fragment"><div class="line">    <span class="keyword">static</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga6395ca871a678020c4a31fadf7e8cc63">cv::Ptr&lt;cv::dnn::Layer&gt;</a> create(<a class="code" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html">cv::dnn::LayerParams</a>&amp; <a class="code" href="../../d1/dae/namespacecv_1_1gapi_1_1ie.html#a3ab1729bcaf2d08e30dd2bc645410908">params</a>);</div></div><!-- fragment --><p> This method should create an instance of you layer and return <a class="el" href="../../dc/d84/group__core__basic.html#ga6395ca871a678020c4a31fadf7e8cc63">cv::Ptr</a> with it.</p>
<ul>
<li>Output blobs' shape computation</li>
</ul>
<div class="fragment"><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">bool</span> getMemoryShapes(<span class="keyword">const</span> std::vector&lt;std::vector&lt;int&gt; &gt; &amp;inputs,</div><div class="line">                                 <span class="keyword">const</span> <span class="keywordtype">int</span> requiredOutputs,</div><div class="line">                                 std::vector&lt;std::vector&lt;int&gt; &gt; &amp;outputs,</div><div class="line">                                 std::vector&lt;std::vector&lt;int&gt; &gt; &amp;internals) <span class="keyword">const</span> <a class="code" href="../../db/de0/group__core__utils.html#ga4d89d63e402ef9ddc48e18e21180fe4a">CV_OVERRIDE</a>;</div></div><!-- fragment --><p> Returns layer's output shapes depends on input shapes. You may request an extra memory using <code>internals</code>.</p>
<ul>
<li>Run a layer</li>
</ul>
<div class="fragment"><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">void</span> forward(<a class="code" href="../../d4/d32/classcv_1_1__InputArray.html">cv::InputArrayOfArrays</a> inputs,</div><div class="line">                         <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> outputs,</div><div class="line">                         <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> internals) <a class="code" href="../../db/de0/group__core__utils.html#ga4d89d63e402ef9ddc48e18e21180fe4a">CV_OVERRIDE</a>;</div></div><!-- fragment --><p> Implement a layer's logic here. Compute outputs for given inputs.</p>
<dl class="section note"><dt>Note</dt><dd>OpenCV manages memory allocated for layers. In the most cases the same memory can be reused between layers. So your <code>forward</code> implementation should not rely that the second invocation of <code>forward</code> will has the same data at <code>outputs</code> and <code>internals</code>.</dd></dl>
<ul>
<li>Optional <code>finalize</code> method</li>
</ul>
<div class="fragment"><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">void</span> finalize(<a class="code" href="../../d4/d32/classcv_1_1__InputArray.html">cv::InputArrayOfArrays</a> inputs,</div><div class="line">                          <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> outputs) <a class="code" href="../../db/de0/group__core__utils.html#ga4d89d63e402ef9ddc48e18e21180fe4a">CV_OVERRIDE</a>;</div></div><!-- fragment --><p> The chain of methods are the following: OpenCV deep learning engine calls <code>create</code> method once then it calls <code>getMemoryShapes</code> for an every created layer then you can make some preparations depends on known input dimensions at <a class="el" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a070e355e782f14f97879f18747fd70fc" title="Computes and sets internal parameters according to inputs, outputs and blobs. ">cv::dnn::Layer::finalize</a>. After network was initialized only <code>forward</code> method is called for an every network's input.</p>
<dl class="section note"><dt>Note</dt><dd>Varying input blobs' sizes such height or width or batch size you make OpenCV reallocate all the internal memory. That leads efficiency gaps. Try to initialize and deploy models using a fixed batch size and image's dimensions.</dd></dl>
<h2>Example: custom layer from Caffe</h2>
<p>Let's create a custom layer <code>Interp</code> from <a href="https://github.com/cdmh/deeplab-public">https://github.com/cdmh/deeplab-public</a>. It's just a simple resize that takes an input blob of size <code>N x C x Hi x Wi</code> and returns an output blob of size <code>N x C x Ho x Wo</code> where <code>N</code> is a batch size, <code>C</code> is a number of channels, <code>Hi x Wi</code> and <code>Ho x Wo</code> are input and output <code>height x width</code> correspondingly. This layer has no trainable weights but it has hyper-parameters to specify an output size.</p>
<p>In example, </p><div class="fragment"><div class="line">layer {</div><div class="line">  name: &quot;output&quot;</div><div class="line">  type: &quot;Interp&quot;</div><div class="line">  bottom: &quot;input&quot;</div><div class="line">  top: &quot;output&quot;</div><div class="line">  interp_param {</div><div class="line">    height: 9</div><div class="line">    width: 8</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --><p>This way our implementation can look like:</p>
<div class="fragment"><div class="line"><span class="keyword">class </span>InterpLayer : <span class="keyword">public</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html">cv::dnn::Layer</a></div><div class="line">{</div><div class="line"><span class="keyword">public</span>:</div><div class="line">    InterpLayer(<span class="keyword">const</span> <a class="code" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html">cv::dnn::LayerParams</a> &amp;<a class="code" href="../../d1/dae/namespacecv_1_1gapi_1_1ie.html#a3ab1729bcaf2d08e30dd2bc645410908">params</a>) : Layer(params)</div><div class="line">    {</div><div class="line">        outWidth = params.<a class="code" href="../../d9/d2b/classcv_1_1dnn_1_1Dict.html#ac5a19bb3d7c245a8a33d596016ed9919">get</a>&lt;<span class="keywordtype">int</span>&gt;(<span class="stringliteral">&quot;width&quot;</span>, 0);</div><div class="line">        outHeight = params.<a class="code" href="../../d9/d2b/classcv_1_1dnn_1_1Dict.html#ac5a19bb3d7c245a8a33d596016ed9919">get</a>&lt;<span class="keywordtype">int</span>&gt;(<span class="stringliteral">&quot;height&quot;</span>, 0);</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">static</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga6395ca871a678020c4a31fadf7e8cc63">cv::Ptr&lt;cv::dnn::Layer&gt;</a> create(<a class="code" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html">cv::dnn::LayerParams</a>&amp; <a class="code" href="../../d1/dae/namespacecv_1_1gapi_1_1ie.html#a3ab1729bcaf2d08e30dd2bc645410908">params</a>)</div><div class="line">    {</div><div class="line">        <span class="keywordflow">return</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga6395ca871a678020c4a31fadf7e8cc63">cv::Ptr&lt;cv::dnn::Layer&gt;</a>(<span class="keyword">new</span> InterpLayer(params));</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">bool</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a3405ff43150088f3cc1d54f7e9b3ec29">getMemoryShapes</a>(<span class="keyword">const</span> std::vector&lt;std::vector&lt;int&gt; &gt; &amp;inputs,</div><div class="line">                                 <span class="keyword">const</span> <span class="keywordtype">int</span> requiredOutputs,</div><div class="line">                                 std::vector&lt;std::vector&lt;int&gt; &gt; &amp;outputs,</div><div class="line">                                 std::vector&lt;std::vector&lt;int&gt; &gt; &amp;internals) <span class="keyword">const</span> CV_OVERRIDE</div><div class="line">    {</div><div class="line">        CV_UNUSED(requiredOutputs); CV_UNUSED(internals);</div><div class="line">        std::vector&lt;int&gt; outShape(4);</div><div class="line">        outShape[0] = inputs[0][0];  <span class="comment">// batch size</span></div><div class="line">        outShape[1] = inputs[0][1];  <span class="comment">// number of channels</span></div><div class="line">        outShape[2] = outHeight;</div><div class="line">        outShape[3] = outWidth;</div><div class="line">        outputs.assign(1, outShape);</div><div class="line">        <span class="keywordflow">return</span> <span class="keyword">false</span>;</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="comment">// Implementation of this custom layer is based on https://github.com/cdmh/deeplab-public/blob/master/src/caffe/layers/interp_layer.cpp</span></div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">void</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a39236126b72a365b94b132abd5bd946b">forward</a>(<a class="code" href="../../d4/d32/classcv_1_1__InputArray.html">cv::InputArrayOfArrays</a> inputs_arr,</div><div class="line">                         <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> outputs_arr,</div><div class="line">                         <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> internals_arr) CV_OVERRIDE</div><div class="line">    {</div><div class="line">        <span class="keywordflow">if</span> (inputs_arr.depth() == <a class="code" href="../../d1/d1b/group__core__hal__interface.html#ga9d2ee1a8334733dea7482a47a88e0f87">CV_16S</a>)</div><div class="line">        {</div><div class="line">            <span class="comment">// In case of DNN_TARGET_OPENCL_FP16 target the following method</span></div><div class="line">            <span class="comment">// converts data from FP16 to FP32 and calls this forward again.</span></div><div class="line">            <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#ae240acf2b7ad43531ca903c927334c8a">forward_fallback</a>(inputs_arr, outputs_arr, internals_arr);</div><div class="line">            <span class="keywordflow">return</span>;</div><div class="line">        }</div><div class="line"></div><div class="line">        std::vector&lt;cv::Mat&gt; inputs, outputs;</div><div class="line">        inputs_arr.getMatVector(inputs);</div><div class="line">        outputs_arr.getMatVector(outputs);</div><div class="line"></div><div class="line">        <a class="code" href="../../d3/d63/classcv_1_1Mat.html">cv::Mat</a>&amp; inp = inputs[0];</div><div class="line">        <a class="code" href="../../d3/d63/classcv_1_1Mat.html">cv::Mat</a>&amp; out = outputs[0];</div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">float</span>* inpData = (<span class="keywordtype">float</span>*)inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a4d33bed1c850265370d2af0ff02e1564">data</a>;</div><div class="line">        <span class="keywordtype">float</span>* outData = (<span class="keywordtype">float</span>*)out.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a4d33bed1c850265370d2af0ff02e1564">data</a>;</div><div class="line"></div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">int</span> batchSize = inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>[0];</div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">int</span> numChannels = inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>[1];</div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">int</span> inpHeight = inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>[2];</div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">int</span> inpWidth = inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>[3];</div><div class="line"></div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">float</span> rheight = (outHeight &gt; 1) ? static_cast&lt;float&gt;(inpHeight - 1) / (outHeight - 1) : 0.f;</div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">float</span> rwidth = (outWidth &gt; 1) ? static_cast&lt;float&gt;(inpWidth - 1) / (outWidth - 1) : 0.f;</div><div class="line">        <span class="keywordflow">for</span> (<span class="keywordtype">int</span> h2 = 0; h2 &lt; outHeight; ++h2)</div><div class="line">        {</div><div class="line">            <span class="keyword">const</span> <span class="keywordtype">float</span> h1r = rheight * h2;</div><div class="line">            <span class="keyword">const</span> <span class="keywordtype">int</span> h1 = <span class="keyword">static_cast&lt;</span><span class="keywordtype">int</span><span class="keyword">&gt;</span>(h1r);</div><div class="line">            <span class="keyword">const</span> <span class="keywordtype">int</span> h1p = (h1 &lt; inpHeight - 1) ? 1 : 0;</div><div class="line">            <span class="keyword">const</span> <span class="keywordtype">float</span> h1lambda = h1r - h1;</div><div class="line">            <span class="keyword">const</span> <span class="keywordtype">float</span> h0lambda = 1.f - h1lambda;</div><div class="line">            <span class="keywordflow">for</span> (<span class="keywordtype">int</span> w2 = 0; w2 &lt; outWidth; ++w2)</div><div class="line">            {</div><div class="line">                <span class="keyword">const</span> <span class="keywordtype">float</span> w1r = rwidth * w2;</div><div class="line">                <span class="keyword">const</span> <span class="keywordtype">int</span> w1 = <span class="keyword">static_cast&lt;</span><span class="keywordtype">int</span><span class="keyword">&gt;</span>(w1r);</div><div class="line">                <span class="keyword">const</span> <span class="keywordtype">int</span> w1p = (w1 &lt; inpWidth - 1) ? 1 : 0;</div><div class="line">                <span class="keyword">const</span> <span class="keywordtype">float</span> w1lambda = w1r - w1;</div><div class="line">                <span class="keyword">const</span> <span class="keywordtype">float</span> w0lambda = 1.f - w1lambda;</div><div class="line">                <span class="keyword">const</span> <span class="keywordtype">float</span>* pos1 = inpData + h1 * inpWidth + w1;</div><div class="line">                <span class="keywordtype">float</span>* pos2 = outData + h2 * outWidth + w2;</div><div class="line">                <span class="keywordflow">for</span> (<span class="keywordtype">int</span> c = 0; c &lt; batchSize * numChannels; ++c)</div><div class="line">                {</div><div class="line">                    pos2[0] =</div><div class="line">                      h0lambda * (w0lambda * pos1[0] + w1lambda * pos1[w1p]) +</div><div class="line">                      h1lambda * (w0lambda * pos1[h1p * inpWidth] + w1lambda * pos1[h1p * inpWidth + w1p]);</div><div class="line">                    pos1 += inpWidth * inpHeight;</div><div class="line">                    pos2 += outWidth * outHeight;</div><div class="line">                }</div><div class="line">            }</div><div class="line">        }</div><div class="line">    }</div><div class="line"></div><div class="line"><span class="keyword">private</span>:</div><div class="line">    <span class="keywordtype">int</span> outWidth, outHeight;</div><div class="line">};</div></div><!-- fragment --><p> Next we need to register a new layer type and try to import the model.</p>
<div class="fragment"><div class="line">    <a class="code" href="../../df/d8c/layer_8details_8hpp.html#a7e8d9c0c5849b6a081ba2a84845f3dac">CV_DNN_REGISTER_LAYER_CLASS</a>(Interp, InterpLayer);</div><div class="line">    <a class="code" href="../../db/d30/classcv_1_1dnn_1_1Net.html">cv::dnn::Net</a> caffeNet = <a class="code" href="../../d6/d0f/group__dnn.html#ga3b34fe7a29494a6a4295c169a7d32422">cv::dnn::readNet</a>(<span class="stringliteral">&quot;/path/to/config.prototxt&quot;</span>, <span class="stringliteral">&quot;/path/to/weights.caffemodel&quot;</span>);</div></div><!-- fragment --> <h2>Example: custom layer from TensorFlow</h2>
<p>This is an example of how to import a network with <a href="https://www.tensorflow.org/versions/master/api_docs/python/tf/image/resize_bilinear">tf.image.resize_bilinear</a> operation. This is also a resize but with an implementation different from OpenCV's or <code>Interp</code> above.</p>
<p>Let's create a single layer network: </p><div class="fragment"><div class="line">inp = tf.placeholder(tf.float32, [2, 3, 4, 5], <span class="stringliteral">&#39;input&#39;</span>)</div><div class="line">resized = tf.image.resize_bilinear(inp, size=[9, 8], name=<span class="stringliteral">&#39;resize_bilinear&#39;</span>)</div></div><!-- fragment --><p> OpenCV sees that TensorFlow's graph in the following way:</p>
<div class="fragment"><div class="line">node {</div><div class="line">  name: &quot;input&quot;</div><div class="line">  op: &quot;Placeholder&quot;</div><div class="line">  attr {</div><div class="line">    key: &quot;dtype&quot;</div><div class="line">    value {</div><div class="line">      type: DT_FLOAT</div><div class="line">    }</div><div class="line">  }</div><div class="line">}</div><div class="line">node {</div><div class="line">  name: &quot;resize_bilinear/size&quot;</div><div class="line">  op: &quot;Const&quot;</div><div class="line">  attr {</div><div class="line">    key: &quot;dtype&quot;</div><div class="line">    value {</div><div class="line">      type: DT_INT32</div><div class="line">    }</div><div class="line">  }</div><div class="line">  attr {</div><div class="line">    key: &quot;value&quot;</div><div class="line">    value {</div><div class="line">      tensor {</div><div class="line">        dtype: DT_INT32</div><div class="line">        tensor_shape {</div><div class="line">          dim {</div><div class="line">            size: 2</div><div class="line">          }</div><div class="line">        }</div><div class="line">        tensor_content: &quot;\t\000\000\000\010\000\000\000&quot;</div><div class="line">      }</div><div class="line">    }</div><div class="line">  }</div><div class="line">}</div><div class="line">node {</div><div class="line">  name: &quot;resize_bilinear&quot;</div><div class="line">  op: &quot;ResizeBilinear&quot;</div><div class="line">  input: &quot;input:0&quot;</div><div class="line">  input: &quot;resize_bilinear/size&quot;</div><div class="line">  attr {</div><div class="line">    key: &quot;T&quot;</div><div class="line">    value {</div><div class="line">      type: DT_FLOAT</div><div class="line">    }</div><div class="line">  }</div><div class="line">  attr {</div><div class="line">    key: &quot;align_corners&quot;</div><div class="line">    value {</div><div class="line">      b: false</div><div class="line">    }</div><div class="line">  }</div><div class="line">}</div><div class="line">library {</div><div class="line">}</div></div><!-- fragment --><p> Custom layers import from TensorFlow is designed to put all layer's <code>attr</code> into <a class="el" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html" title="This class provides all data needed to initialize layer. ">cv::dnn::LayerParams</a> but input <code>Const</code> blobs into <a class="el" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a9a5578e0b3a0ec0301fb7320b54aa6ed" title="List of learned parameters must be stored here to allow read them by using Net::getParam(). ">cv::dnn::Layer::blobs</a>. In our case resize's output shape will be stored in layer's <code>blobs[0]</code>.</p>
<div class="fragment"><div class="line"><span class="keyword">class </span>ResizeBilinearLayer <a class="code" href="../../db/de0/group__core__utils.html#ga8b49a79bdb8458a658db563481a19f4e">CV_FINAL</a> : <span class="keyword">public</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html">cv::dnn::Layer</a></div><div class="line">{</div><div class="line"><span class="keyword">public</span>:</div><div class="line">    ResizeBilinearLayer(<span class="keyword">const</span> <a class="code" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html">cv::dnn::LayerParams</a> &amp;<a class="code" href="../../d1/dae/namespacecv_1_1gapi_1_1ie.html#a3ab1729bcaf2d08e30dd2bc645410908">params</a>) : Layer(params)</div><div class="line">    {</div><div class="line">        <a class="code" href="../../db/de0/group__core__utils.html#gaf62bcd90f70e275191ab95136d85906b">CV_Assert</a>(!params.<a class="code" href="../../d9/d2b/classcv_1_1dnn_1_1Dict.html#ac5a19bb3d7c245a8a33d596016ed9919">get</a>&lt;<span class="keywordtype">bool</span>&gt;(<span class="stringliteral">&quot;align_corners&quot;</span>, <span class="keyword">false</span>));</div><div class="line">        <a class="code" href="../../db/de0/group__core__utils.html#gaf62bcd90f70e275191ab95136d85906b">CV_Assert</a>(!blobs.empty());</div><div class="line"></div><div class="line">        <span class="keywordflow">for</span> (<span class="keywordtype">size_t</span> i = 0; i &lt; blobs.size(); ++i)</div><div class="line">            <a class="code" href="../../db/de0/group__core__utils.html#gaf62bcd90f70e275191ab95136d85906b">CV_Assert</a>(blobs[i].<a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a5f3cdb2524e281094e14e212a76a2d38">type</a>() == <a class="code" href="../../d1/d1b/group__core__hal__interface.html#ga32f03fbb8f73bff70215b77f5c3cac11">CV_32SC1</a>);</div><div class="line"></div><div class="line">        <span class="comment">// There are two cases of input blob: a single blob which contains output</span></div><div class="line">        <span class="comment">// shape and two blobs with scaling factors.</span></div><div class="line">        <span class="keywordflow">if</span> (blobs.size() == 1)</div><div class="line">        {</div><div class="line">            <a class="code" href="../../db/de0/group__core__utils.html#gaf62bcd90f70e275191ab95136d85906b">CV_Assert</a>(blobs[0].<a class="code" href="../../df/d57/namespacecv_1_1dnn.html#a65ad6cf1b64a572bf78d696d2014b0e6">total</a>() == 2);</div><div class="line">            outHeight = blobs[0].at&lt;<span class="keywordtype">int</span>&gt;(0, 0);</div><div class="line">            outWidth = blobs[0].at&lt;<span class="keywordtype">int</span>&gt;(0, 1);</div><div class="line">            factorHeight = factorWidth = 0;</div><div class="line">        }</div><div class="line">        <span class="keywordflow">else</span></div><div class="line">        {</div><div class="line">            <a class="code" href="../../db/de0/group__core__utils.html#gaf62bcd90f70e275191ab95136d85906b">CV_Assert</a>(blobs.size() == 2); <a class="code" href="../../db/de0/group__core__utils.html#gaf62bcd90f70e275191ab95136d85906b">CV_Assert</a>(blobs[0].<a class="code" href="../../df/d57/namespacecv_1_1dnn.html#a65ad6cf1b64a572bf78d696d2014b0e6">total</a>() == 1); <a class="code" href="../../db/de0/group__core__utils.html#gaf62bcd90f70e275191ab95136d85906b">CV_Assert</a>(blobs[1].<a class="code" href="../../df/d57/namespacecv_1_1dnn.html#a65ad6cf1b64a572bf78d696d2014b0e6">total</a>() == 1);</div><div class="line">            factorHeight = blobs[0].at&lt;<span class="keywordtype">int</span>&gt;(0, 0);</div><div class="line">            factorWidth = blobs[1].at&lt;<span class="keywordtype">int</span>&gt;(0, 0);</div><div class="line">            outHeight = outWidth = 0;</div><div class="line">        }</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">static</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga6395ca871a678020c4a31fadf7e8cc63">cv::Ptr&lt;cv::dnn::Layer&gt;</a> create(<a class="code" href="../../db/db6/classcv_1_1dnn_1_1LayerParams.html">cv::dnn::LayerParams</a>&amp; params)</div><div class="line">    {</div><div class="line">        <span class="keywordflow">return</span> <a class="code" href="../../dc/d84/group__core__basic.html#ga6395ca871a678020c4a31fadf7e8cc63">cv::Ptr&lt;cv::dnn::Layer&gt;</a>(<span class="keyword">new</span> ResizeBilinearLayer(params));</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">bool</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a3405ff43150088f3cc1d54f7e9b3ec29">getMemoryShapes</a>(<span class="keyword">const</span> std::vector&lt;std::vector&lt;int&gt; &gt; &amp;inputs,</div><div class="line">                                 <span class="keyword">const</span> <span class="keywordtype">int</span>,</div><div class="line">                                 std::vector&lt;std::vector&lt;int&gt; &gt; &amp;outputs,</div><div class="line">                                 std::vector&lt;std::vector&lt;int&gt; &gt; &amp;) const CV_OVERRIDE</div><div class="line">    {</div><div class="line">        std::vector&lt;int&gt; outShape(4);</div><div class="line">        outShape[0] = inputs[0][0];  <span class="comment">// batch size</span></div><div class="line">        outShape[1] = inputs[0][1];  <span class="comment">// number of channels</span></div><div class="line">        outShape[2] = outHeight != 0 ? outHeight : (inputs[0][2] * factorHeight);</div><div class="line">        outShape[3] = outWidth != 0 ? outWidth : (inputs[0][3] * factorWidth);</div><div class="line">        outputs.assign(1, outShape);</div><div class="line">        <span class="keywordflow">return</span> <span class="keyword">false</span>;</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">void</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a070e355e782f14f97879f18747fd70fc">finalize</a>(<a class="code" href="../../d4/d32/classcv_1_1__InputArray.html">cv::InputArrayOfArrays</a>, <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> outputs_arr) CV_OVERRIDE</div><div class="line">    {</div><div class="line">        std::vector&lt;cv::Mat&gt; outputs;</div><div class="line">        outputs_arr.getMatVector(outputs);</div><div class="line">        <span class="keywordflow">if</span> (!outWidth &amp;&amp; !outHeight)</div><div class="line">        {</div><div class="line">            outHeight = outputs[0].size[2];</div><div class="line">            outWidth = outputs[0].size[3];</div><div class="line">        }</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="comment">// This implementation is based on a reference implementation from</span></div><div class="line">    <span class="comment">// https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/kernels/internal/reference/reference_ops.h</span></div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">void</span> <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#a39236126b72a365b94b132abd5bd946b">forward</a>(<a class="code" href="../../d4/d32/classcv_1_1__InputArray.html">cv::InputArrayOfArrays</a> inputs_arr,</div><div class="line">                         <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> outputs_arr,</div><div class="line">                         <a class="code" href="../../d2/d9e/classcv_1_1__OutputArray.html">cv::OutputArrayOfArrays</a> internals_arr) CV_OVERRIDE</div><div class="line">    {</div><div class="line">        <span class="keywordflow">if</span> (inputs_arr.depth() == <a class="code" href="../../d1/d1b/group__core__hal__interface.html#ga9d2ee1a8334733dea7482a47a88e0f87">CV_16S</a>)</div><div class="line">        {</div><div class="line">            <span class="comment">// In case of DNN_TARGET_OPENCL_FP16 target the following method</span></div><div class="line">            <span class="comment">// converts data from FP16 to FP32 and calls this forward again.</span></div><div class="line">            <a class="code" href="../../d3/d6c/classcv_1_1dnn_1_1Layer.html#ae240acf2b7ad43531ca903c927334c8a">forward_fallback</a>(inputs_arr, outputs_arr, internals_arr);</div><div class="line">            <span class="keywordflow">return</span>;</div><div class="line">        }</div><div class="line"></div><div class="line">        std::vector&lt;cv::Mat&gt; inputs, outputs;</div><div class="line">        inputs_arr.getMatVector(inputs);</div><div class="line">        outputs_arr.getMatVector(outputs);</div><div class="line"></div><div class="line">        <a class="code" href="../../d3/d63/classcv_1_1Mat.html">cv::Mat</a>&amp; inp = inputs[0];</div><div class="line">        <a class="code" href="../../d3/d63/classcv_1_1Mat.html">cv::Mat</a>&amp; out = outputs[0];</div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">float</span>* inpData = (<span class="keywordtype">float</span>*)inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a4d33bed1c850265370d2af0ff02e1564">data</a>;</div><div class="line">        <span class="keywordtype">float</span>* outData = (<span class="keywordtype">float</span>*)out.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a4d33bed1c850265370d2af0ff02e1564">data</a>;</div><div class="line"></div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">int</span> batchSize = inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>[0];</div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">int</span> numChannels = inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>[1];</div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">int</span> inpHeight = inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>[2];</div><div class="line">        <span class="keyword">const</span> <span class="keywordtype">int</span> inpWidth = inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>[3];</div><div class="line"></div><div class="line">        <span class="keywordtype">float</span> heightScale = <span class="keyword">static_cast&lt;</span><span class="keywordtype">float</span><span class="keyword">&gt;</span>(inpHeight) / outHeight;</div><div class="line">        <span class="keywordtype">float</span> widthScale = <span class="keyword">static_cast&lt;</span><span class="keywordtype">float</span><span class="keyword">&gt;</span>(inpWidth) / outWidth;</div><div class="line">        <span class="keywordflow">for</span> (<span class="keywordtype">int</span> b = 0; b &lt; batchSize; ++b)</div><div class="line">        {</div><div class="line">            <span class="keywordflow">for</span> (<span class="keywordtype">int</span> y = 0; y &lt; outHeight; ++y)</div><div class="line">            {</div><div class="line">                <span class="keywordtype">float</span> input_y = y * heightScale;</div><div class="line">                <span class="keywordtype">int</span> y0 = <span class="keyword">static_cast&lt;</span><span class="keywordtype">int</span><span class="keyword">&gt;</span>(std::floor(input_y));</div><div class="line">                <span class="keywordtype">int</span> y1 = <a class="code" href="../../d7/dcc/group__core__utils__softfloat.html#gac48df53b8fd34b87e7b121fa8fd4c379">std::min</a>(y0 + 1, inpHeight - 1);</div><div class="line">                <span class="keywordflow">for</span> (<span class="keywordtype">int</span> x = 0; x &lt; outWidth; ++x)</div><div class="line">                {</div><div class="line">                    <span class="keywordtype">float</span> input_x = x * widthScale;</div><div class="line">                    <span class="keywordtype">int</span> x0 = <span class="keyword">static_cast&lt;</span><span class="keywordtype">int</span><span class="keyword">&gt;</span>(std::floor(input_x));</div><div class="line">                    <span class="keywordtype">int</span> x1 = <a class="code" href="../../d7/dcc/group__core__utils__softfloat.html#gac48df53b8fd34b87e7b121fa8fd4c379">std::min</a>(x0 + 1, inpWidth - 1);</div><div class="line">                    <span class="keywordflow">for</span> (<span class="keywordtype">int</span> c = 0; c &lt; numChannels; ++c)</div><div class="line">                    {</div><div class="line">                        <span class="keywordtype">float</span> interpolation =</div><div class="line">                            inpData[offset(inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>, c, x0, y0, b)] * (1 - (input_y - y0)) * (1 - (input_x - x0)) +</div><div class="line">                            inpData[offset(inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>, c, x0, y1, b)] * (input_y - y0) * (1 - (input_x - x0)) +</div><div class="line">                            inpData[offset(inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>, c, x1, y0, b)] * (1 - (input_y - y0)) * (input_x - x0) +</div><div class="line">                            inpData[offset(inp.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>, c, x1, y1, b)] * (input_y - y0) * (input_x - x0);</div><div class="line">                        outData[offset(out.<a class="code" href="../../d3/d63/classcv_1_1Mat.html#a146f8e8dda07d1365a575ab83d9828d1">size</a>, c, x, y, b)] = interpolation;</div><div class="line">                    }</div><div class="line">                }</div><div class="line">            }</div><div class="line">        }</div><div class="line">    }</div><div class="line"></div><div class="line"><span class="keyword">private</span>:</div><div class="line">    <span class="keyword">static</span> <span class="keyword">inline</span> <span class="keywordtype">int</span> offset(<span class="keyword">const</span> <a class="code" href="../../df/d63/structcv_1_1MatSize.html">cv::MatSize</a>&amp; <a class="code" href="../../df/d5b/namespacecv_1_1gapi_1_1streaming.html#a0a915e69f4cc8284293e40fc9ffbf157">size</a>, <span class="keywordtype">int</span> c, <span class="keywordtype">int</span> x, <span class="keywordtype">int</span> y, <span class="keywordtype">int</span> b)</div><div class="line">    {</div><div class="line">        <span class="keywordflow">return</span> x + size[3] * (y + size[2] * (c + size[1] * b));</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keywordtype">int</span> outWidth, outHeight, factorWidth, factorHeight;</div><div class="line">};</div></div><!-- fragment --><p> Next we register a layer and try to import the model.</p>
<div class="fragment"><div class="line">    <a class="code" href="../../df/d8c/layer_8details_8hpp.html#a7e8d9c0c5849b6a081ba2a84845f3dac">CV_DNN_REGISTER_LAYER_CLASS</a>(ResizeBilinear, ResizeBilinearLayer);</div><div class="line">    <a class="code" href="../../db/d30/classcv_1_1dnn_1_1Net.html">cv::dnn::Net</a> tfNet = <a class="code" href="../../d6/d0f/group__dnn.html#ga3b34fe7a29494a6a4295c169a7d32422">cv::dnn::readNet</a>(<span class="stringliteral">&quot;/path/to/graph.pb&quot;</span>);</div></div><!-- fragment --> <h2>Define a custom layer in Python</h2>
<p>The following example shows how to customize OpenCV's layers in Python.</p>
<p>Let's consider <a href="https://arxiv.org/abs/1504.06375">Holistically-Nested Edge Detection</a> deep learning model. That was trained with one and only difference comparing to a current version of <a href="http://caffe.berkeleyvision.org/">Caffe framework</a>. <code>Crop</code> layers that receive two input blobs and crop the first one to match spatial dimensions of the second one used to crop from the center. Nowadays Caffe's layer does it from the top-left corner. So using the latest version of Caffe or OpenCV you'll get shifted results with filled borders.</p>
<p>Next we're going to replace OpenCV's <code>Crop</code> layer that makes top-left cropping by a centric one.</p>
<ul>
<li>Create a class with <code>getMemoryShapes</code> and <code>forward</code> methods</li>
</ul>
<div class="fragment"><div class="line"><span class="keyword">class </span>CropLayer(object):</div><div class="line">    <span class="keyword">def </span>__init__(self, params, blobs):</div><div class="line">        self.xstart = 0</div><div class="line">        self.xend = 0</div><div class="line">        self.ystart = 0</div><div class="line">        self.yend = 0</div><div class="line"></div><div class="line">    <span class="comment"># Our layer receives two inputs. We need to crop the first input blob</span></div><div class="line">    <span class="comment"># to match a shape of the second one (keeping batch size and number of channels)</span></div><div class="line">    <span class="keyword">def </span>getMemoryShapes(self, inputs):</div><div class="line">        inputShape, targetShape = inputs[0], inputs[1]</div><div class="line">        batchSize, numChannels = inputShape[0], inputShape[1]</div><div class="line">        height, width = targetShape[2], targetShape[3]</div><div class="line"></div><div class="line">        self.ystart = (inputShape[2] - targetShape[2]) // 2</div><div class="line">        self.xstart = (inputShape[3] - targetShape[3]) // 2</div><div class="line">        self.yend = self.ystart + height</div><div class="line">        self.xend = self.xstart + width</div><div class="line"></div><div class="line">        <span class="keywordflow">return</span> [[batchSize, numChannels, height, width]]</div><div class="line"></div><div class="line">    <span class="keyword">def </span>forward(self, inputs):</div><div class="line">        <span class="keywordflow">return</span> [inputs[0][:,:,self.ystart:self.yend,self.xstart:self.xend]]</div></div><!-- fragment --> <dl class="section note"><dt>Note</dt><dd>Both methods should return lists.</dd></dl>
<ul>
<li>Register a new layer.</li>
</ul>
<div class="fragment"><div class="line">cv.dnn_registerLayer(<span class="stringliteral">&#39;Crop&#39;</span>, CropLayer)</div></div><!-- fragment --><p> That's it! We've replaced an implemented OpenCV's layer to a custom one. You may find a full script in the <a href="https://github.com/opencv/opencv/tree/master/samples/dnn/edge_detection.py">source code</a>.</p>
<table border="0">
<tr>
<td><div class="image">
<img src="../../lena.jpg" alt="lena.jpg"/>
</div>
 </td><td><div class="image">
<img src="../../lena_hed.jpg" alt="lena_hed.jpg"/>
</div>
  </td></tr>
</table>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:34 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
